While the human eye has a unique ability to adapt to a wide range of brightness levels ranging from a bright sunny day down to single photon levels, conventional image sensors, such as complementary metal-oxide semiconductor (CMOS) and charge-coupled device (CCD) array image sensors (note that CMOS and CCD pixels are sometimes referred to herein as conventional pixels, and an array of such conventional pixels is sometimes referred to herein as a conventional image sensor), convert incident light into an electrical signal proportional to the brightness of the scene point up to a saturation point. This conversion process in conventional image sensors cannot simultaneously capture very dark and very bright regions that occur in many natural scenes. These limitations are often apparent in images of scenes captured using consumer products such as digital cameras incorporated into smartphones. However, these limitations can have additional effects in more specialized applications. For example, a vision sensor used in an autonomous vehicle driving at night may encounter scenes that include bright light sources (e.g., headlights of oncoming vehicles) while the rest of the scene remains relatively dark. As another example, an industrial robot using a machine vision system to assemble metallic machine parts may capture images that include strong specular highlights and shadows on the metallic parts which may exceed the relatively low dynamic range of many conventional image sensors. As yet another example, an image sensor used for microscope imaging may capture an image of a back-lit sample with high contrast, which may exceed the dynamic range of a conventional image sensor. In such examples, bright regions in the scene can cause conventional sensor pixels to saturate, whereas dark regions of the scene can be dominated by noise.
One approach to increasing dynamic range in conventional image sensors is to capture more light from darker parts of a scene to average out sensor noise, and capture less light from brighter parts of the scene to avoid saturation. A common computational technique called exposure bracketing uses multiple images of the scene captured using different exposure times, and blends the pixel values to generate a high dynamic range (HDR) image. This technique works relatively well for static scenes, but suffers from ghosting artifacts when capturing dynamic scenes that contain relatively fast-moving objects. Single-shot HDR sensors were developed in part to avoid these artifacts, and requires changing the image sensor hardware to include optical elements that modulate the amount of light reaching the sensor pixels. For example, some single-shot HDR sensors use fixed or adaptive light absorbing neutral density filters placed in front of the sensor pixels to spatially vary the amount of light received during a fixed exposure time. As another example, some single-shot HDR sensors use beam-splitters to relay the scene onto multiple imaging sensors with different exposure settings.
Other HDR imaging approaches involve redesigning the sensor pixel hardware to obtain a non-linear relationship between scene radiance (sometimes also referred to herein as brightness) and sensor output. For example, logarithmic image sensors use additional hardware in each pixel that applies logarithmic non-linearity to obtain dynamic range compression. While the approaches described above provide greater dynamic range than conventional image sensors, the dynamic range that can be achieved is limited by the hard saturation limit of the pixels.
Quanta image sensors (QIS) are a relatively new approach to HDR imaging that attempt to improve dynamic range through spatial oversampling. These QIS sensors exploit fine-grained (i.e., sub-diffraction-limit) spatial statistics to improve dynamic range. A QIS array is constructed with millions of pixels of sub-diffraction limit sizes, with each pixel being a binary sensor with negligible read noise. This enables designing a response function that mimics a silver-halide photographic plate with a logarithmic response curve that provides some overexposure latitude and avoids a hard saturation limit. However, gains in dynamic range using QIS techniques have been limited due to difficulties in manufacturing pixels smaller than about a micron in size. As discussed below in connection with FIG. 16, to achieve very high dynamic range using a QIS sensor, pixel size would need to be reduced by at least an order of magnitude (e.g., to at most 100 nm) which is not feasible using current technology.
Accordingly, new systems, methods, and media for high dynamic range imaging are desirable.